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Abstract 

This paper describes an instrument designed for assessing learning outcomes in data management. In 
addition to assessment of student learning and ABET outcomes, we have also found the instrument to 
be effective for determining database placement of incoming information systems (IS) graduate 
students. Each of these three uses is discussed in this paper. We describe the use of a pre/post test, 
item validation, and correlation techniques for the purpose of validation and assessment. Although 
the instrument was developed for local assessment, its design is based on international information 
systems curriculum guidelines rendering it suitable for use in any program which incorporates 
database management in its curriculum. 
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1. INTRODUCTION 

Universities are increasingly being required to 
demonstrate that student learning is occurring 
at their institutions in measurable, documented 
ways, and that these measurable results are 
being used to improve their educational 
programs. Assessment of learning has become 
a requirement of institutional and program 
accreditation. Many methods of assessment are 
possible, including internally/externally 
developed, direct/indirect measures of 
performance, and formative/summative 


indicators. Often these assessment approaches 
are developed for "local" use, i.e. they are not 
designed to be generalized for use by similar 
programs at peer institutions. This paper 
describes the development, validation, use, and 
results interpretation of a database exam—an 
internally-developed, direct assessment, 
formative indicator of student learning in a four- 
year information systems (IS) degree program— 
that we believe can be used for assessment in 
any program requiring a database management 
course. In the sections that follow, we describe 
the foundation for the exam, the approach taken 
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for developing and verifying exam items, the 
approach taken for validating that the exam is a 
useful instrument for student outcomes 
assessment, and a discussion of the several uses 
that we have made of the instrument. 

2. BACKGROUND 

The exam was developed in the mid-2000's as 
an outgrowth of a national certification exam 
project, and for use at the co-authors' 
university—the University of South Alabama 
(USA), located in Mobile, Alabama. Available 
from the Center for Forensics, Information 
Technology, and Security, the USA-CFITS DB 
Exam consists of 25 multiple choice items, 16 of 
which appear on the IS 2002 exit exam, a 
national certification exam for information 
systems exit skills (Landry, Reynolds, & 
Longenecker, & 2003). 

The original reason for creating the exam was to 
address a graduate program placement issue. 
Students admitted to the information systems 
master's program had traditionally been placed 
into the graduate data management course 
based on the prerequisite of having passed an 
undergraduate database course. Despite having 
transcript evidence of an undergraduate 
database management course at other 
institutions, some students were not prepared to 
succeed in our graduate database course. Since 
our undergraduate course was designed to 
satisfy course objectives consistent with learning 
units in IS 2002 and since graduate students 
who successfully completed our undergraduate 
database course also successfully completed the 
graduate database course, we concluded that a 
placement exam was needed to accurately 
determine when the undergraduate course 
should be a required prerequisite. 
Subsequently, the database placement exam 
was created to be given to incoming master's 
students, and used as a placement mechanism. 
Students making a passing score were admitted 
to the graduate data management course, while 
students making a failing score were advised to 
complete the undergraduate database course 
with a passing grade of 'C' or better. 

Development and Validation of the Exam 

The USA-CFITS DB Exam was originally designed 
to be a measure of data management knowledge 
and skills, one of the fundamental core areas of 
Information Systems curricula (Landry, 
Longenecker, Haigood, & Feinstein, 2000; 


11 ( 2 ) 
April 2013 


Haigood 2001; Colvin 2008). The foundations for 
the exam are database-related learning units 
(LU) of IS curricula models, IS'90, IS'97, and 
IS2002 (Longenecker & Feinstein, 1991; 
Longenecker, Feinstein, Couger, Davis, & 
Gorgone, 1995; Davis, Gorgone, Couger, 
Feinstein, & Longenecker, 1997; Gorgone, 
Davis, Valacich, Topi, Feinstein, & Longenecker, 
2003). The continuing relevance of database 
skills and knowledge in the IS curricula models 
is further supported by the results of two 
surveys—one targeting faculty and industry 
partners (Landry et al., 2000) and a second 
targeting IS professionals two to four years 
beyond graduation (Colvin, 2008). 

Specific knowledge and skill areas used to 
motivate item writing for the USA-CFITS DB 
Exam were drawn from prior work reflecting an 
intersection of academic and professional needs. 
Henderson, Champlin, Coleman, Cupoli, Hoffer, 
Howarth, Sivier, Smith, & Smith (2004) 
published a framework for Data Management 
curricula intended for postsecondary education 
and sponsored by a professional society, the 
Data Management Association (DAMA). 
Longenecker, Henderson, Smith, Cupoli, 
Yarbrough, Smith, Gillenson, & Feinstein (2006) 
studied this framework in detail and found that 
the skills were compatible with the IS2002 and 
IS2010 IS curriculum guidelines. Table 5 in the 
appendix reflects a synthesis of the DAMA 
framework, the IS model curriculum guidelines, 
and a job ad analysis (Landry et al., 2000; 
Haigood 2001). 

In developing the USA-CFITS DB Exam to reflect 
both professional skills and curriculum 
guidelines, the authors wrote items that 
assessed the intersection of a data management 
sub-skill area and an IS 2002 learning unit. The 
learning objectives for each of the 25 items on 
the USA-CFITS DB Exam are as follows: 

1. Given a piece of data to programmatically 
manipulate, choose the appropriate data 
type 

2. Given a real-world application, determine 
appropriate fields to be stored in a file 

3. Choose and defend the correct data type for 
representing a common data attribute 

4. Differentiate between entities and attributes 
when developing an ERD 

5. Recognize the need either for an intersection 
table in a M:N relationship or the need to 
revisit requirements to determine if there is 
a missing entity 
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6. Given a relational database description, 
evaluate the architecture 

7. Given a system need, such as access control 
to a database, identify the necessary 
information 

8. Differentiate among alternatives for 
enforcing data integrity constraints 

9. Compare and contrast the processes 
involved in data modeling 

10. Recognize the implication of a cascade 
delete 

11. Recognize the notation of standard ER 
models 

12. Recognize and describe a correct three- 
entity solution to a problem expressed as a 
many-to-many relationship between two 
entities 

13. Recognize that many-to-many relationships 
require a third, linking table in a relational 
DB 

14. Apply the knowledge of using a stored 
procedure to enhance the performance in a 
database environment 

15. Given database design goals, identify correct 
techniques for implementation 

16. Normalize (redesign) an unnormalized 
(poorly designed) table 

17. Recognize correct syntax and correct use of 
views 

18. Recognize the implication of using views in a 
client application 

19. Recognize the advantages and 
disadvantages of implementation with stored 
procedures 

20. Trace and debug SQL syntax 

21. Recognize the correct formulation of a query 

22. Differentiate normal forms as part of 
database design 

23. Recognize which tasks are associated with 
discovering and eliciting database design 
requirements in the initial phase of 
requirements analysis 

24. Recognize relevant factors involved in the 
purchasing decision of a major enterprise 
level DBMS package 

25. Recognize properties of the Entity- 
Relationship Model, particularly the concept 
of minimum cardinality 

Since the development of the USA-CFITS DB 
Exam, a revision of the information systems 
curriculum guidelines has been issued. IS 2010, 
available at 

http://www.acm.org/education/curricula . defines 
core course IS 2010.2 as Data and Information 
Management. All 25 USA-CFITS DB Exam items 
map to a stated course objective of the IS 
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2010.2 course. Of the 25 items, 13 of them map 
to course objectives 6, 8, and 12, dealing with 
conceptual data modeling, designing a high 
quality database, and various SQL commands, 
and 13 of the 21 course objectives are covered 
by at least one exam item. 

The exam item objectives were also mapped to 
ABET student outcomes criteria (ABET, 2007, p. 
14). The outcomes criteria, along with the 
number of exam items mapped to each, are 
shown in Table 1. See Table 5 in the appendix 
for a grand mapping of the 25 item objectives 
with IS 2002, IS 2010 and ABET. 


Table 1 - Coverage of ABET Student 
Outcomes 



Student Outcomes 
that must be 
enabled 

Number of 
associated 
exam item 
obiectives 


(a) An ability to 
apply knowledge of 
computing and 
mathematics 
appropriate to the 
discipline 

1 

The program 
has 

documented 
measurable 
outcomes 
that are 
based on the 
needs of the 
program's 
constituencies 

(b) An ability to 
analyze a problem, 
and identify and 
define the 
computing 
requirements 
appropriate to its 
solution 

5 

(c) An ability to 
design, implement 
and evaluate a 
computer-based 
system, process, 
component, or 
program to meet 
desired needs 

12 



(i) An ability to use 
current techniques, 
skills, and tools 
necessary for 
computinq 

7 


It is important that an internal exam designed 
for assessment be mappable into multiple 
assessment frameworks. Doing so strengthens 
the validity of the exam's content as being 
relevant outside of the local unit's needs. For 
more on the approach used to map multiple 
assessment frameworks, write items, and 
validate exams, see related papers (Landry et 
al., 2003; Landry , Daigle, Longenecker, & 
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Pardue, 2010; Reynolds, Longenecker, Landry, 
Pardue, & Applegate, 2004). 

Exam Construction 

The multiple mappings established a useful 
foundation for item writing, which was carried 
out using these and other good practices in 
educational assessment (Hogan 2007; Crocker & 
Algina 1986). The writers wrote items and 
objectives in alignment with mapped 
frameworks. An item consisted of a stem with 
four possible answers with one correct answer. 
Good item writing was difficult, and multiple 
reviewers were utilized in the item review 
process. The entire item-writing and review 
process was supported by a web-based exam 
delivery system developed by the co-authors 
and their graduate students at the University of 
South Alabama. The candidate items were pilot 
tested, revised, and validated with statistical 
techniques, including test item statistics. See 
Section 3 - Validation below for details. A 
summary of recommended practices includes the 
following: 

• Define objectives, and write items that 
target the objectives 

• Map items into other outcomes for 
assessment value 

• Don't write items that are too difficult 

• Make sure items are based on knowledge 

• Get multiple reviewers to rigorously review 
items, and correct 

• Pilot test the exam 

• Use test item statistics to validate 

• Make exam easy to administer and score 

• Select an appropriate passing score 

• Develop good security policies 

See Figure 1 for an overview of the item 
construction process. 

A cut score for passing was set at 44% correct 
responses. The success rate of students in our 
graduate database course correlated with 
whether the student made at least a 44. A score 
of 44 correlated with a midrange 'C' 
performance in our undergraduate database 
course. While the score of 44 would seem low 
for a student who has taken a database 
management course, an explanation is that 
scores for this external exam are predictably 
lower than scores on internal assessments that 
reflect an individual instructor's preferences in 


11 ( 2 ) 
April 2013 


instructional approach and topic emphasis. 
Furthermore, we designed the items on the 
exam to be discriminating, that is, to 
differentiate between those who know and those 
who don't, perhaps to a higher degree than 
instructors do in general. 



Figure 1 - Item Construction Process 


Multiple Uses of the Exam 

The faculty eventually found multiple uses for 
the exam in addition to graduate data 
management course placement. In the 
undergraduate database course, the exam is 
given as a pre-test at the beginning of the 
course and as a post-test incorporated as part of 
the final exam. This practice provides the 
capability of assessing the degree to which the 
undergraduate database course is achieving its 
intended learning outcomes, independent of 
instructor assignment (especially part-time 
instructors) and in different delivery formats 
(traditional, blended, fully online). This results 
are used as a formative program assessment 
method for both ABET and regional accreditation 
agencies (e.g. SACS). 
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3. VALIDATION 

The results of using the exam over three years 
are described next. The first test described is a 
test using content experts. This test was 
intended as a face validity test, but also 
demonstrated content validity. The panel of 
experts, which consisted of professors from the 
university using the exam, took the test as a 
student would, in a proctored lab environment. 

Overall, observations made by the experts 
included a perception that the test items are 
discriminating, that is, they are effective at 
discriminating between whether someone knew 
the answer or would have to guess. The 
perception among the content experts is 
testable. See discussion of item validation and 
pre/post testing below. Another positive 

reaction from an expert after taking the test was 
that "I knew what the item was about, but don't 
know if I got it right." This comment was 
interpreted as meaning the item was about a 
relevant database concept familiar to the expert, 
but that the item was also challenging. Another 
expert said that it was helpful that the exam had 
a consistent format of diagrams and tables that 
accompanied some of the items, as well as re¬ 
use of data in tables. Such consistency cuts 
down on the cognitive overload on takers. The 
eight items (of 25) that use tables or figures 
depict ER models, queries, or tables/views of 
data. One expert liked the "normalization item", 
another liked the item on "intersection tables" 
(which table gets the foreign key?"). 

More critically, the experts thought that "four or 
five items need revisiting (more review)." Some 
jargon was recognized as being potentially 
confusing to students, including the use of 
United States zip codes on a data types item. 
The toughest items were believed to be those on 
triggers and constraints. The experts were 
skeptical of items that presumed a specific order 
of database life cycle activities. Another item 
asked about the "best way" to do something, 
and it was believed the item to be too 
normative. 

The second set of tests we conducted was to run 
statistical analyses on the most recent set of 
test taker data. We calculated summary and 
item statistics, and conducted pre/post tests, 
and ran correlations of test vs. course 
performance. 
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Summary and Test Item Statistics 

From January 2008 until May 2010, a total of 
246 USA students, a combination of graduate 
and undergraduate students, English speaking 
and ESL students, took the USA-CFITS DB 
Exam. Over this period, 53.4 was the mean 
score with standard deviation of 14.6. This 
score is consistent with national norms for the 
information systems exit exam. The highest 
score was a 92, and the lowest score was a 16. 
Eight test takers, or a little more than 3 percent 
of all takers, scored below 25, or worse than 
guessing. 

The KR20, which measures internal item 
consistency, was 0.62. The score is right above 
a minimally acceptable score of 0.60, which is 
recommended for tests in a subject domain 
taken by those trained in that domain. 


Table 2 - Item Statistics 


Pet Correct 

Point Biserial 

43 

0.45 

64 

0.36 

58 

0.24 

65 

0.46 

40 

0.40 

50 

0.51 

80 

0.30 

54 

0.26 

58 

0.25 

34 

0.20 

40 

0.12 

81 

0.41 

75 

0.43 

86 

0.19 

32 

0.34 

58 

0.14 

72 

0.26 

28 

0.21 

87 

0.29 

30 

0.51 

39 

0.36 

53 

0.34 

26 

0.30 

28 

0.30 

46 

0.44 


Some test item statistics are provided in Table-2 
below. This table indicates the percentage of 
subjects getting each item correct, which varies 
from 26% to 87%, and the point biserial, which 
varies from .12 to .51. The percent correct 
scores indicate item difficulty on a 100-point 
scale, with a 100 representing the easiest (least 
difficult) item, that is, with 100% of takers 
answering it correctly. Higher point biserials are 
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indicative of items that correlate well with the 
exam as whole, especially when values are 0.40 
and higher. 

Pre and Post tests 

The purpose of a pre/post test is to demonstrate 
that learning took place between the two 
measurements. In our case, we gave the USA- 
CFITS DB Exam to incoming graduate students. 
Those (25 students) who failed to make a 
passing score were required to take an 
undergraduate database course, and three other 
students who barely passed also decided to take 
the database course. 


Table 3 - Pre/Post Test Results 


Taker # 

Pre¬ 

test 

score 

Post¬ 

test 

score 

Difference 
b/w pre & 
post 

1 

24 

52 

28 

2 

32 

48 

16 

3 

36 

56 

20 

4 

28 

52 

24 

5 

16 

56 

40 

6 

40 

56 

16 

7 

28 

60 

32 

8 

36 

68 

32 

9 

40 

76 

36 

10 

48 

68 

20 

11 

44 

68 

24 

12 

32 

44 

12 

13 

24 

44 

20 

14 

40 

48 

8 

15 

40 

48 

8 

16 

20 

40 

20 

17 

40 

48 

8 

18 

32 

32 

0 

19 

64 

72 

8 

20 

24 

56 

32 

21 

40 

68 

28 

22 

36 

36 

0 

23 

32 

48 

16 

24 

32 

44 

12 

25 

40 

52 

12 

26 

40 

60 

20 

27 

40 

56 

16 

28 

36 

44 

8 





# Failed 

25 

3 


# Passed 

3 

25 


Total 

takers 

28 

28 


Pet takers 
passed 

11% 

89% 


Mean 

score (0- 
100) 

35.1 

53.6 

18.4 
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At the end of the database course, they again 
took the placement exam. These two sets of 
scores were compared using a paired t-test, 
using PASW Statistics. There were 28 students 
in the sample. The pre/post test scores are in 
Table 3 as follows. 

By the end of the course the results were 
reversed. There were now 25 passing scores 
and three that were still below passing (although 
one of those improved by 20 points) for a pass 
rate of 89%. The pre-test mean was 35.1, 
compared to a post-test mean of 53.6. The 
mean difference was 18.4 points, and the result 
of a paired differences test was statistically 
significant at a .001 level (p=.000). Such a 
result is a strong indicator of learning taking 
place in the course. It was particularly 
remarkable that the increase in scores occurred 
despite the fact that many of the students in the 
sample had prior database experience and 
scored close to passing in the pre-test. 

If the test maps well to the objectives of the 
course, and the pre-test is given to those with 
little knowledge of the subject matter, a 
pre/post test design ought to detect whether 
learning is taking place. In this way, we can use 
the USA-CFITS DB Exam to verify that the 
undergraduate course is achieving its planned 
learning outcomes, over time, especially as the 
instructor changes. Once a pre/post relationship 
is established, it might be sufficient just to give 
the post-test, and compare the post test mean 
to historical post-test averages. 

Correlations of test taker performance vs. 
database course performance 

Over time (see Table 4), we determined that the 
scores on the exam correlated as follows: 

Table 4 - Exam-Course Correlations 


Score on 
USA-CFITS 

DB Exam (% 
correct) 

Associated 
letter grade 
in the course 

60-100 

A 

50-59 

B 

40-49 

C 

30-39 

D 

0-29 

F 


The grading scale on an exam like this is not the 
same as a typical 10-point scale used commonly 
in universities, with 90-100 A, 80-89 B, etc. The 


©2013 EDSIG (Education Special Interest Group of the AITP) 
www.aitp-edsig.org /www.isedj.org 


Page 68 
















































Information Systems Education Journal (ISEDJ) 
ISSN: 1545-679X 


items on the exam, while representative of a 
first database course, are not particular to a 
specific institution's database course or its 
instructor. 

We believe that instructors taught the database 
course in an unbiased manner towards the 
exam. It should be noted that that data 
includes scores from students in sections taught 
by two of the co-authors, one of whom also 
developed questions for this exam. The co¬ 
author's approach in teaching the course was 
not to teach to the test, nor use exam items 
elsewhere in the course. The other instructors 
had no access to the exam items before, during, 
and after the pre/post tests. 

4. CONCLUSION 

In summary, the benefits of using the exam are 
as follows: 

• Maps to ABET outcomes 

• Provides instructor-independent assessment 
of learning 

• Can use as a placement exam for grad 
program or transfer students 

• Useful for outcomes assessment for ABET 
accreditation 

• Useful for course assessment 

With the growing demand for more outcomes- 
based assessment in higher education, the use 
of this type of internally-developed exam, while 
becoming necessary, will offer many benefits. 
Among these are instructor-independent course 
and program outcomes assessment that 
supports multiple frameworks. We have shown 
that the USA-CFITS DB Exam is aligned with 
international curriculum models, ABET outcomes 
and job-related skills from two surveys (Landry 
et al., 2000; Colvin, 2008). With the specific 
exam being described, the USA-CFITS DB Exam, 
we have provided evidence that success in a first 
database course is most closely correlated with 
mastery of a specific subset of learning 
outcomes in data management. We described 
how we were able to converge on a cut score 
that predicted whether or not a graduate 
student needed to take a database prerequisite 
course. We provided evidence that post-test 
student scores parallel their local course 
performance, while trending lower than local 
scores for predictable reasons (i.e. exam is not 
specific to an instructor or the local course). All 
this made the exam useful for student 
placement and course assessment. 
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We believe that the need for more and better 
assessment helps make efforts like ours 
worthwhile. To inquire about use of the exam, 
contact the University of South Alabama Center 
for Forensics, Information Technology, and 
Security (USA-CFITS, http://www.usacfits.org ). 
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